NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm

Neural networks (NNs) have begun to have a pervasive impact on various applications of machine learning. However, the problem of finding an optimal NN architecture for large applications has remained open for several decades. Conventional approaches search for the optimal NN architecture through extensive trial-and-error. Such a procedure is quite inefficient. In addition, the generated NN architectures incur substantial redundancy. To address these problems, we propose an NN synthesis tool (NeST) that automatically generates very compact architectures for a given dataset. NeST starts with a seed NN architecture. It iteratively tunes the architecture with gradient-based growth and magnitudebased pruning of neurons and connections. Our experimental results show that NeST yields accurate yet very compact NNs with a wide range of seed architecture selection. For example, for the LeNet-300-100 (LeNet-5) NN architecture derived from the MNIST dataset, we reduce network parameters by 34.1× (74.3×) and floating-point operations (FLOPs) by 35.8× (43.7×). For the AlexNet NN architecture derived from the ImageNet dataset, we reduce network parameters by 15.7× and FLOPs by 4.6×. All these results are the current state-of-the-art for these architectures.